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PROJECT  GOALS  AND  OBJECTIVES 

The  PI  is  examining  atmospheric  predictability  with  the  goal  of  improving  ensemble  forecasts  at 
ranges  of  12  hours  to  10  days.  The  research  is  addressing  several  issues,  including: 

♦>  Documentation  of  analysis  uncertainty  from  mesoscale  and  global  analyses. 

♦♦♦  Calibration  of  ensemble  forecast  system  (EFS)  output  by  artificial  neural  networks. 

♦♦♦  Design  of  optimal  EFS's,  with  an  emphasis  on  precipitation  forecasts. 

♦>  Design  of  stochastic  physics  parameterizations  that  improve  under-dispersion  in  EFS’s. 

The  PI  also  served  as  Co-Chief  Scientist  to  Dr.  Scott  Sandgathe  for  ONR  initiative  on  Predictability  in 
the  Atmosphere  and  Ocean,  and  presumably  will  continue  to  serve  as  Co-Chief  Scientist  on  the 
initiative. 

DOCUMENTATION  OF  ANALYSIS  UNCERTAINTY 

The  PI  is  estimating  lower  bounds  for  the  statistics  of  analysis  errors  Eo  from  differences  between 
different  analysis-forecast  systems.  This  approach  defines  a  “component"  of  the  analysis  uncertainty. 
Although  this  methodology  is  not  as  theoretically  appealing  as  statistics  from  ensemble  data 
assimilation  or  observing  system  simulation  experiments,  it  is  currently  tractable,  very  economical,  and 
useful  guidance  can  be  quickly  obtained. 

The  PI  has  been  comparing  differences  among  NCEP  LAM  analyses  from  the  NGM  and  ETA  models 
and  the  global  AVN  model.  It  is  important  to  consider  LAM  fields  since  scales  not  resolved  by  global 
analyses  are  presumably  analyzed  with  greater  certainty  over  the  data  rich,  North  American  continent. 
Approximately  three  years  of  twice-daily  analyses  have  been  collected  processed.  In  FY01,  the  PI 
began  analysis  of  the  data,  with  the  computation  of  2D  spectra  and  vertical  correlation  coefficients  of 
the  difference  fields  to  document  the  scale  dependency  of  analysis  uncertainty.  The  analysis  has 
progressed  little  over  the  past  year,  exception  for  the  collection  of  another  year  of  data.  PI  is 
collaborating  with  DRI  participants  Errico,  Baumhefner  and  Tribbia  (NCAR)  to  document  analysis 
uncertainty  in  global  analyses.  Analyses  from  ECMWF  and  NCEP  are  being  compared,  and  similar 
statistics  are  being  computed  for  the  global  difference  fields.  This  work  is  the  natural  complement  to 
the  LAM  documentation.  A  2D  spectrum  is  shown  in  Fig.  lfor  500  mb  temperatures.  The  variance  for 
the  difference  between  analyses  is  as  large  as  the  variance  for  ECMWF  fields  for  wavenumbers  higher 
than  36,  which  indicates  global  analyses  of  temperature  contain  little  useful  information  for 
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wavelengths  shorter  than  10°  latitude  (-1,100  km).  The  beginnings  of  a  manuscript  are  in  early  stages 
of  preparation. 
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Fig.  1.  Two-dimensional  variance  spectra  for  500  mb  temperatures  for  the  1991 
boreal  winter.  Spectrum  for  differences  between  ECMWF-NCEP  global  analyses 
(red)  and  for  ECMWF  analyses  (black).  The  difference  variance  equals  or  exceeds 
the  analysis  variance  beyond  wavenumber  36. 


NEURAL  NETWORK  POST-PROCESSING  OF  ENSEMBLE  FORECAST  PRODUCTS 

Because  forecast  fields  produced  by  any  NWP  model  always  contain  errors  due  to  model  deficiencies 
(e.g.  lack  of  resolution,  inadequate  parameterizations,  truncation  error,  etc),  raw  model  output  is  often 
statistically  post-processed  to  mitigate  their  impact.  Post-processing  also  provides  a  way  to  relate 
model  output  fields  to  weather  elements  not  explicitly  forecast  by  the  NWP  model  (e.g.  visibility, 
probability  of  thunder). 

Artificial  neural  networks  (ANNs)  are  computer  algorithms  that  are  designed  for  nonlinear 
optimization.  A  back-propagation  ANN  was  used  to  process  to  QPF  output  from  the  pilot  ETA/RSM 
ensemble  data  set  that  was  also  used  by  Hamill  and  Colucci  (1998).  The  ANN  markedly  improves  the 
unprocessed  ensemble  and  even  shows  higher  skill  than  Model  Output  Statistics,  the  de  facto 
operational  standard  based  on  multiple  linear  regression,  for  thresholds  up  to  1.00".  The  improvement 
comes  from  a  better  reliability,  the  forecast  probability  matching  the  observed  frequency  of  occurrence, 
but  forecast  specificity,  the  ability  to  pick  up  day-to-day  cases  in  event  occurrence,  did  not  improve. 
These  results  will  be  reported  in  a  paper  (Mullen  et  al.  2002,  still  in  preparation). 

DESIGN  OF  OPTIMAL  ENSEMBLE  FORECAST  SYSTEMS 

The  PI  and  collaborator  R.  Buizza  (ECMWF)  examined  the  tradeoff  between  increased  horizontal 
resolution  and  ensemble  size  on  ensemble  performance  for  precipitation.  Experiments  at  resolutions  of 
T159,  T255  and  T319  were  analyzed.  Potential  values  as  a  function  of  the  Cost-Loss  ratio  at  Day  5  for 
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Fig.  2.  Potential  economic  value,  as  a  function  of  a  user’s  Cost-Loss  ratio,  for  four  configurations 
of  the  ECMWF  ensemble  prediction  system  (EPS).  The  102-member  ensemble  (T159+T255M102), 
comprised  of  two  51-member  lower-resolution  models  (T159M51  and  T255M51),  provides  higher 
value  than  the  51-member  ensemble  at  higher  resolution  (T319M51).  The  comparison  indicates  that 
increased  ensemble  size  can  yield  higher  value  for  many  users  than  higher  resolution. 


amounts  above  50  mm  in  24  h  are  shown  in  Fig.  2.  Values  are  shown  for  four  ensemble  configurations: 
three  51 -member  ensembles  at  truncations  of  T 159,  T255  and  T319,  and  a  102-member  ensemble 
comprised  of  the  T159  and  T255  constructs.  Note  that  the  102-member  ensemble  takes  less  CPU  time 
than  the  T3 19M5 1  ensemble.  Our  analysis  indicates  that  increased  ensemble  size  can  provide  better 
value  than  increasing  the  model  resolution,  especially  for  rare  (in  terms  of  climatological  likelihood  of 
occurrence)  events.  The  issue  of  how  to  invest  CPU  resource  needs  to  be  explored  for  more  ensemble 
systems,  various  thresholds,  etc.  An  in-press  paper  (Mullen  and  Buizza,  2002)  reports  these  results. 

In  collaboration  with  scientists  at  the  NOAA  National  Severe  Storms  Laboratory,  the  value  of  mixed- 
ensemble  forecast  systems  for  QPF  and  PQPF  is  being  explored  for  model  resolutions  comparable  to 
the  current  22-km  ETA  and  ranges  out  to  36  h.  Fig.  3  shows  a  three-panel  plot  with  the  area  under  the 
ROC  curve  (which  measures  the  ability  of  a  forecast  system  to  discriminate  events)  and  parameters  for 
potential  economic  value.  The  panels  give  four  thresholds  for  24  h  accumulations  ending  at  36  h  for  a 
9-member  mixed  ensemble.  ROC  areas  are  positive  for  all  thresholds.  This  indicates  that  hit  rate  for 
the  ensemble  exceeds  that  false  alann  rate.  The  Max  Value  plot  gives  the  maximum  economic  value; 
the  panel  shows  that  the  ensemble  has  value  relative  to  climatology  for  even  the  1.00”  threshold.  The 
Value  Width  panel  gives  the  interval  of  Cost/Loss  ratios  with  positive  values.  Together,  the  Max  Value 
and  Value  Width  panels  show  that  mixed  ensembles  offer  value  to  many  decision  makers.  These 
results  are  reported  in  a  recent  paper  (Wandishin  et  al  2001). 
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Under  ONR  support  in  FY02,  the  PI  will  continue  examining  optimal  configurations  for  ensemble 
forecast  systems.  The  PI  plans  to  explore  predictability  limits  and  the  utility  of  high  membership 
ensembles  0(100)  from  daily  forecasts  with  a  mesoscale  LAM  (~10  km  grid)  with  hardware  acquired 
under  DURIP  support.  The  new  hardware  is  ordered  and  is  scheduled  for  delivery  in  Oct.  2002. 
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Fig.  3.  Summary  of  cost-loss,  value  for  24  It  rainfall  totals  valid  at  +36  It  from  a 
mixed-physics  ensemble  forecast  system.  Upper  Left:  Maximum  Value  for  four 
thresholds.  Upper  Right:  Width  of  interval  with  positive  Cost-Loss  ratios.  Lower 
Right:  Standardized  area  under  the  Relative  Operating  Characteristic  curve.  The 
results  show  that  mesoscale  ensemble  systems  can  provide  useful  guidance  to 
decision  makers,  but  that  value  drops  rapidly  with  increased  precipitation  threshold. 


DESIGN  OF  STOCHASTIC  MODEL  PARAMETERIZATION S 

In  collaboration  with  David  Bright  (PhD  Candidate  in  ATMO),  the  feasibility  of  stochastic  cumulus 
parameterization  schemes  is  being  tested  with  a  LAM.  The  goal  is  the  design  schemes  that  give 
performance  comparable  to  mixed-physics  ensemble  systems.  The  MM5  system  and  the  Kain-Fritsch 
cumulus  scheme  are  being  used  to  investigate  perturbations  to  the  scheme's  trigger  function.  Estimates 
of  the  life  span  of  cumulus  convection  from  radar  data  were  used  to  model  the  temporal  correlation  as 
a  1st  order  auto-regressive  AR(1)  process  with  an  e-folding  time  of  1  hr.  Weak,  spatial  correlations 
implicitly  come  from  the  fact  that  the  AR(1)  process  evolves  slowly  relative  to  the  model  time  step. 
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Figure  4  (next  page)  shows  an  interesting  result:  the  scheme  leads  to  upscale  error  growth.  Note  the 
horizontal-scale  of  the  precipitation  perturbations  by  +24  h  is  clearly  larger  than  the  smallest  resolvable 
"28x"  wavelength  (72-km)  in  the  model.  This  indicates  the  AR(1)  temporal  correlation  can  yield  non¬ 
trivial,  spatially-correlated  perturbations. 

Figure  5  summarizes  the  potential  economic  value  for  a  25mm/24h  threshold  and  +24  h  forecast. 
Results  for  four  10-member,  ensembles  are  shown:  a  12  km  ensemble  with  Kain-Fritsch  and  MRF  PBL 
schemes  with  only  perturbed  analyses  (12KM  label),  a  36  km  ensemble  with  KF  and  MRF  schemes 
with  only  perturbed  analyses  (K-M  label),  a  perturbed-analyses  mixed-model  36  km  ensemble  based 
on  10  unique  combinations  of  Cu  schemes  and  PBL  schemes  (MIX  label),  and  a  36  km  stochastic  KF 
and  deterministic  MRF  scheme  (K’-M  label).  Note  only  one  parameter  in  the  KF  scheme  is  stochastic, 
the  trigger  function.  The  plot  shows  positive  value  for  all  configurations.  For  most  users,  the  MIX  is 
superior  and  the  K-M  inferior.  The  12KM  gives  a  slight  improvement  over  the  K-M.  Note  that  the  K’- 
M  lies  about  halfway  between  the  MIX  and  K-M  lines.  In  another  words,  the  single  stochastic 
parameter  K’-M  scheme  yields  ~50%  of  the  benefit  of  a  mixed-model,  perturbed-analyses  scheme. 


Fig.  4.  Difference  between  a  stochastic  physics  forecast  and  the  control  forecast  without  stochastic 
physics  for  24  h  rainfall  totals  valid  at  +24  h.  Differences  occur  at  scales  longer  than  twice  the  grid 
point  spacing,  which  indicates  the  occurrence  of  upscale  error  growth. 


5 


Fig.  5.  Potential  Value  as  function  of  a  hypothetical  user’s  Cost/Lost  Ratio  for  4  different  ensemble 
configurations.  The  mixed-physics  ensemble  (MIX)  provides  the  greatest  potential  value  to  most 
users.  The  Kain-Fritsch  with  MRF  PBL  scheme  (K-M)  provides  the  least  value  to  most  users.  A 
stochastic  version  of  the  Kain-Fritsch  cumulus  scheme  with  the  MRF  PBL  package  (K’-M)  provides 
better  value  than  a  12  km  version  (12KM)  of  the  K-M  ensemble.  The  results  show  that  including  a 
stochastic  representation  for  one  process  in  the  Kain-Fritsch  scheme  can  provide  about  half  the 

increased  value  for  a  full  mixed-physics  ensemble. 
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